CLAIMS 

We claim: 

1 . A method in a computing system for tracking the movement of 
a dimensional surface having an arbitrary appearance relative to a camera, 
comprising: 

capturing an image of the surface; 

analyzing the captured image of the surface to identify visual 
features present in the captured image; 

from the identified visual features, selecting a plurality of visual 
features for use in tracking the movement of the surface; 

receiving a sequence of images captured by the camera, at least 
some of which constitute a view of at least a portion of the surface; 

for each image of the sequence: 

identifying the 2-dimensional positions at which the selected 
features occur in the image; and 

based upon the 2-dimensional positions at which the features 
are identified in the image, determining the 3-dimensional location and orientation 
of the surface in the current image of the sequence with respect to the camera. 

2. The method of claim 1 wherein the identification of the 2- 
dimensional positions at which the selected features occur in the image is 
predicated on an assumption that the selected features appear as coplanar in the 
image. 

3. The method of claim 1 wherein the surface is a 2-dimensional 

surface. 

4. The method of claim 1 wherein the surface is a flat surface. 
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5. The method of claim 1 wherein the surface is an irregular 
body that appears flat when observed at a distance. 

6. The method of claim 1, further comprising using the 
determined location and orientation of the surface to introduce a supplemental 
image into the images of the sequence at a size, location, and orientation that are 
relative to those of the surface. 

7. The method of claim 1, further comprising using the 
determined location and orientation of the surface to superimpose a view of a 3- 
dimensional object over the surface. 

8. The method of claim 1 wherein the selection of identified 
features is performed based upon a comparison of the levels of contrast provided 
by each of the identified features. 

9. The method of claim 1 wherein the selection of identified 
features is performed based upon a comparison of the levels of uniqueness of 
each of the identified features among all of the identified features. 

10. The method of claim 1 wherein the selection of identified 
features is performed based upon a comparison of the levels of accuracy with 
which they can be used to determine the position and orientation of the surface. 

11. The method of claim 1 wherein the selection of identified 
features includes selecting visual features in at least two different size ranges, 

the method further comprising selecting one of the size ranges based upon a 
measure of the distance to the surface's present location, 
and wherein selected features in the selected size range are used to determine 
the location and orientation of the surface. 
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12. The method of claim 1, further comprising, for each image of 
the sequence: 

based upon the 2-dimensional positions at which the features are 
identified in one or more prior images of the sequence, predicting 2-dimensional 
positions at which the features will occur in the current image; and 

establishing search zones about the predicted positions, 
and wherein identifying the 2-dimensional positions at which the selected features 
occur in the image comprises searching the established search zones for the 
selected features. 

13. The method of claim 1 wherein the surface, in addition to 
having an arbitrary appearance, is modified to contain one or more fiducial 
markers, 

the method further comprising, in a first image of the sequence, identifying and 
analyzing a portion of the image corresponding to the fiducial markers to 
determining the 3-dimensional location and orientation of the surface in the image, 
and wherein the determining the 3-dimensional location and orientation of the 
surface in the first image of the sequence is used to identify the 2-dimensional 
positions at which the selected features occur in the first image of the sequence. 

14. The method of claim 13 wherein each fiducial marker 
comprises a unique distinguishing pattern. 

15. The method of claim 13 wherein each fiducial marker is 
square-shaped. 

16. The method of claim 1 wherein the 3-dimensional location and 
orientation of the surface in the current image of the sequence is determined 
without the use of explicit fiducial markers. 
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17. The method of claim 1 further comprising capturing the 
sequence of images using the camera, 

and wherein the determination is made in real-time with respect to the capture. 



18. The method of claim 1 wherein, between two successive 
images of the sequence, the camera moves relative to the environment. 

19. The method of claim 1 wherein, between two successive 
images of the sequence, the surface moves relative to the environment. 

20. A computer-readable medium whose contents cause a 
computing system to track the movement of a dimensional surface having an 
arbitrary appearance relative to a camera by: 

capturing an image of the surface; 

analyzing the captured image of the surface to identify visual 
features present in the captured image; 

from the identified visual features, selecting a plurality of visual 
features for use in tracking the movement of the surface; 

receiving a sequence of images captured by the camera, at least 
some of which constitute a view of at least a portion of the surface; 

for each image of the sequence: 

identifying the 2-dimensional positions at which the selected 
features occur in the image; and 

based upon the 2-dimensional positions at which the features 
are identified in the image, determining the 3-dimensional location and orientation 
of the surface in the current image of the sequence with respect to the camera. 
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21 . A method in a computing system for tracking a 2-dimensional 
surface having an innate visual appearance and being marked with one or more 
explicit fiducial markers, comprising: 

identifying visually significant aspects of the surface's innate visual 

appearance; 

in a first of a series of perspective images of the surface, performing 
recognition of the fiducial markers to determine the 3-dimensional location and 
orientation of the surface in the first image; and 

in successive perspective images of the surface, performing 
recognition of the identified aspects of the surface's innate visual appearance to 
determine the 3-dimensional location and orientation of the surface in the 
successive images. 

22. The method of claim 21 wherein the recognition of the 
identified aspects of the surface's innate visual appearance in each successive 
image is guided by the 3-dimensional location and orientation of the surface in the 
preceding image. 

23. The method of claim 21 wherein each fiducial marker is 
square-shaped. 

24. The method of claim 21 wherein the recognition of the 
identified aspects of the surface's innate visual appearance is predicated on an 
assumption that the selected features appear as coplanar in the successive 
perspective images. 

25. A computing system for tracking a 2-dimensional surface 
having an innate visual appearance and being marked with one or more explicit 
fiducial markers, comprising: 
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a visual analysis subsystem that identifies visually significant aspects 
of the surface's innate visual appearance; 

a first recognition subsystem that, in a first of a series of perspective 
images of the surface, performs recognition of the fiducial markers to determine 
the 3-dimensional location and orientation of the surface in the first image; and 

a second recognition subsystem that, in successive perspective 
images of the surface, performs recognition of the identified aspects of the 
surface's innate visual appearance to determine the 3-dimensional location and 
orientation of the surface in the successive images. 

26. A method in a computing system for determining the 3- 
dimensional location and orientation of a subject surface in a distinguished 
perspective image of the subject surface, the subject surface having innate visual 
features, a subset of which are selected, comprising: 

using the location of the selected visual features in a perspective 
image of the subject surface that precedes the distinguished perspective image in 
time, identifying search zones in the distinguished perspective image; 

searching the identified search zones for the selected visual features 
to determine the 2-dimensional locations at which the selected visual features 
occur; and 

based on the determined 2-dimensional locations, determining the 3- 
dimensional location and orientation of a subject surface in a distinguished 
perspective image. 

27. The method of claim 26 wherein the selected innate visual 
features of the subject surface number at least four. 

28. The method of claim 26 wherein a plurality of subsets of 
innate visual features of the subject surface are selected, each subset containing 
features of a different general size, 
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the method further comprising, based on an earlier-determined 3-dimensional 
location of the subject surface, choosing one subset of features, 
and wherein the identified search zones are searched for the features of the 
chosen subset. 

29. The method of claim 26 wherein the determination of the 2- 
dimensional locations at which the selected visual features occur is predicated on 
an assumption that the selected visual features appear as coplanar in the image. 

30. A computer-readable medium whose contents cause a 
computing system to determine the 3-dimensional location and orientation of a 
subject surface in a distinguished perspective image of the subject surface, the 
subject surface having innate visual features, a subset of which are selected, by: 

using the location of the selected visual features in a perspective 
image of the subject surface that precedes the distinguished perspective image in 
time, identifying search zones in the distinguished perspective image; 

searching the identified search zones for the selected visual features 
to determine the 2-dimensional locations at which the selected visual features 
occur; and 

based on the determined 2-dimensional locations, determining the 3- 
dimensional location and orientation of a subject surface in a distinguished 
perspective image. 

31. One or more computing memories storing a visual tracking 
data structure for a surface having an appearance, the data structure comprising a 
plurality of natural feature templates, each natural feature template corresponding 
to a feature occurring in the appearance of the surface and containing adequate 
information to identify the feature in a perspective image of the surface, 

such that the contents of the data structure may be used to identify at least a 
subset of the features to which the natural feature templates correspond in a 
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perspective image of the surface, and to determine the distance and orientation of 
the surface with respect to the point in space from which the perspective image is 
captured. 
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